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Abstract 

Tuning  to  an  optimal  stimulus  is  a  widespread  property  of  neurons  in  cortex.  We  propose  that  such  tuning 
is  a  consequence  of  normalization  or  gain  control  circuits.  We  also  present  a  biologically  plausible  neural 
circuitry  of  tuning. 
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1  Introduction 

Across  the  cortex,  especially  in  the  sensory  areas,  many 
neurons  respond  strongly  to  some  stimuli,  but  weakly 
to  others,  as  if  they  were  tuned  to  some  optimal  fea¬ 
tures  or  to  particular  input  patterns.  For  example,  neu¬ 
rons  in  primary  visual  cortex  show  Gaussian-like  tun¬ 
ing  in  multiple  dimensions,  such  as  orientation,  spatial 
frequency,  direction,  velocity.  Moving  further  along  the 
ventral  pathway  of  primate  cortex,  V4  neurons  show 
tuned  responses  to  different  types  of  gratings  or  con¬ 
tour  features  [8,  15],  and  some  IT  neurons  are  respon¬ 
sive  to  a  particular  view  of  a  face  or  other  objects 
[11,13]. 

In  other  sensory  modalities,  neural  tuning  is  also 
common.  Olfactory  neurons  in  the  fly  respond  to  par¬ 
ticular  mixtures  of  molecules,  or  odors  [25].  Auditory 
neurons  of  a  song  bird  can  be  tuned  to  sound  patterns, 
or  motif  [9].  In  the  case  of  the  motor  system,  the  activity 
of  a  spinal  cord  neuron  is  related  to  a  particular  pat¬ 
tern  of  force  fields  or  limb  movements  [16].  The  tuning 
of  a  neuron  may  be  sharp  and  sparse  in  some  cases,  or 
distributed  and  general  in  other  cases  [12],  but  despite 
qualitative  differences,  such  tuning  behavior  seems  to 
be  one  of  the  major  computational  strategies  for  repre¬ 
senting  and  encoding  information  in  cortex. 

Consequently,  tuning  in  cortex  is  often  characterized 
and  approximated  with  a  multidimensional  Gaussian 
function  in  many  models.  In  [15],  contour  feature  tun¬ 
ing  in  V4  is  fitted  with  a  Gaussian  function  in  curvature 
and  angular  position  space.  In  [2],  a  similar  Gaussian 
function  is  used  to  characterize  the  response  of  the  af¬ 
ferent  cells  to  the  IT  neurons.  In  the  model  of  visual  ob¬ 
ject  recognition  by  Riesenhuber  and  Poggio,  which  at¬ 
tempts  to  describe  quantitatively  the  first  few  hundred 
milliseconds  of  visual  recognition,  the  Gaussian  func¬ 
tion  is  one  of  the  two  key  operations  for  providing  se¬ 
lectivity  [19]. 

Even  though  Gaussian-like  tuning  behavior  in  cortex 
is  widely  acknowledged,  it  remains  a  major  puzzle  in 
neuroscience:  how  could  such  multidimensional  tun¬ 
ing  be  implemented  by  neurons?  The  underlying  bio¬ 
physical  mechanism  is  not  understood.  In  Hubei  and 
Wiesel's  model  of  VI,  the  tuning  properties  of  simple 
and  complex  cells  are  explained  in  terms  of  the  geom¬ 
etry  of  the  afferents:  for  simple  cells,  the  alignment  of 
several  non-oriented  LGN  afferents  would  give  rise  to 
the  orientation  selectivity  (see  [7]  for  a  review,  and  [21] 
for  a  quantitative  model).  Although  attractively  sim¬ 
ple  and  intuitive,  this  explanation  is  challenged  by  a 
competing  theory  that  maintains  orientation  selectivity 
is  enforced,  if  not  created,  by  the  recurrent  neural  cir¬ 
cuitry  within  VI  [1,  6, 23].  The  tuning  along  non-spatial 
dimensions  such  as  velocity  or  color,  however,  can  not 
rely  on  the  geometric  arrangements  only.  Furthermore, 
tuning  in  other  sensory  modalities  ( e.g .,  auditory  or  ol¬ 
factory  neurons)  and  in  higher  visual  areas  where  the 


tuning  seems  to  be  of  a  more  abstract  nature  (e.g.,  the 
complex  shape  tuning  in  IT)  would  require  a  more  gen¬ 
eral  mechanism. 

In  this  paper,  we  propose  a  biophysically  plausible 
solution  to  the  puzzle  of  Gaussian-like  tuning. 

2  A  general  mechanism  for  cortical  tuning 

As  mentioned  in  the  introduction,  many  neurons  show 
tuning,  which  is  often  described  in  terms  of  a  multidi¬ 
mensional  Gaussian  function: 

f(x)=e  l--^l2/2-2.  (1) 

The  key  operation  in  Eqn.  1  is  \x  —  w\2,  the  computation 
of  similarity  between  two  vectors,  which  determines  a 
tuned  response  around  a  target  vector  w.  However,  we 
do  not  have  any  obvious  neural  circuitry  or  biophys¬ 
ical  mechanism  for  such  operation.  How,  then,  could 
Gaussian-like  tuning  arise  in  cortex? 

One  possible  answer  to  this  puzzle  is  hinted  by  the 
following  mathematical  identity,  which  relates  the  Eu¬ 
clidean  distance  measure,  which  appears  in  the  Gaus¬ 
sian  function,  with  the  normalized  scalar  product: 

\x  —  w\2  =  —2x  ■  w  +  1  +  |xtT|2,  if  |a?|  =  1.  (2) 

In  other  words,  the  similarity  between  two  normalized 
vectors,  x  and  w,  can  be  measured  with  a  Euclidean  dis¬ 
tance  as  well  as  a  scalar  product,  or  the  angle  between 
two  vectors.  Hence,  Eqn.  2  suggests  that  Gaussian-like 
tuning  can  arise  from  a  normalized  scalar  product  op¬ 
eration.* 

The  advantage  of  considering  normalized  scalar 
product  as  a  tuning  operation  is  its  biophysical  plau¬ 
sibility.  Unlike  the  computation  of  Euclidean  distance 
or  a  Gaussian  function,  both  normalization  and  scalar 
product  operations  can  be  readily  implemented  with  a 
network  of  neurons.  The  scalar  product  or  the  weighted 
sum  can  be  computed  by  the  dendritic  inputs  to  a 
cell  with  different  synaptic  weights.  The  normaliza¬ 
tion  across  the  inputs  can  be  achieved  by  a  divisive 
gain  control  mechanism  involving  inhibitory  interac¬ 
tions  [3,  4, 10, 18].  The  neural  response  may  be  subject 
to  extra  nonlinearities,  such  as  sigmoid  or  rectification, 

‘This  relationship  was  pointed  out  by  Maruyama,  Girosi 
and  Poggio  in  [14],  where  the  connection  between  the  mul¬ 
tilayer  perceptron  and  the  neural  network  with  radial  basis 
function  is  explored.  Their  analysis  is  based  on  the  exact  form 
of  this  identity  (i.e.,  the  input  x  to  the  Euclidean  distance  is 
normalized  as  well  as  the  input  to  the  scalar  product).  In  this 
paper,  we  examine  a  looser  connection  between  the  Euclidean 
distance  and  the  normalized  scalar  product  (i.e.,  the  input  to 
the  Euclidean  distance  is  not,  but  the  input  to  the  scalar  prod¬ 
uct  is  normalized): 
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in  the  soma  or  the  axon.  Together,  the  normalized  scalar 
product  with  a  nonlinear  transfer  function  can  give  rise 
to  a  Gaussian-like  tuning  function,  as  shown  in  the  next 
section. 

While  many  neurophysiological  experiments  have 
found  tuning  behaviors  in  cortex,  theoretical  studies 
[16, 17]  also  indicate  that  a  network  based  on  radial  ba¬ 
sis  functions  (like  Gaussian)  indeed  is  a  plausible  com¬ 
putational  scheme  capable  of  learning.  Here,  learning  is 
defined  as  a  capacity  for  generalizing  an  input-output 
mapping  from  a  finite  number  of  data.  In  a  learning 
neural  network  with  radial  basis  functions,  the  "hid¬ 
den"  units  show  Gaussian-like  tuning  behavior  to  the 
input.  More  concretely,  a  computational  model  with  a 
network  of  Gaussian  template-matching  units  is  shown 
to  be  capable  of  performing  object  recognition,  while  re¬ 
producing  the  shape  selectivity  and  invariance  proper¬ 
ties  of  IT  neurons  [19, 20]. 


3  One  plausible  neural  circuit  for  tuning 


Eqn.  2  suggests  that  a  crucial  element  for  tuning  is  the 
normalization,  which  can  be  expressed  mathematically 
as 

Ri  =  r— — -  (3) 

V  lev 


Eqn.  3  can  be  implemented  by  a  pool  of  neurons  N , 
whose  individual  responses  are  divisively  normalized 
by  the  collective  response  across  the  pool,  giving  rise  to 
the  following  two  important  properties. 


1.  Individual  neural  response  is  normalized:  The 

response  of  each  neuron  is  divided  by  the  total 
response  of  the  pool  that  includes  other  neurons 
as  well  as  itself.  The  normalization  factor  is  al¬ 
ways  greater  than  the  numerator  in  Eqn.  3.  Hence, 
the  neural  response  is  upper  bounded  and  oper¬ 
ates  within  a  well-defined  dynamic  range  (i.e.,  Ri  £ 
[0,1])- 


2.  Collective  response  across  the  pool  is  normalized: 

The  sum  of  neural  responses  within  the  normaliza¬ 
tion  pool  is  also  normalized  (i.e.,  Ri  can  be  thought 
of  as  the  ith  component  of  a  normalized  vector  R, 
with  \R\  =  1).  This  aspect  of  normalization  re¬ 
ceived  less  attention  in  the  past,  but  it  may  be  the 
underlying  mechanism  for  cortical  tuning,  which  is 
the  focus  of  this  paper. 

How  would  a  network  of  neurons  accomplish  such 
divisive  normalization  across  the  pool?  In  the  past,  sev¬ 
eral  plausible  neural  circuits  for  gain  control  mecha¬ 
nism  have  been  proposed  and  explored  in  various  con¬ 
texts.  [18]  considered  forward  and  recurrent  shunting 
inhibition  circuits  for  gain  control  within  fly's  visual 
system.  Many  researchers  have  used  the  normalization 


mechanism  to  explain  the  contrast-dependent,  saturat¬ 
ing  neural  responses  in  primary  visual  cortex  [3,  4,  10] 
and  center-surround  effects  within  receptive  field  [5].  In 
[22],  similar  divisive  normalization  scheme  was  shown 
to  increase  independence  of  neural  signals,  despite  the 
dependencies  in  image  statistics. 

Fig.  la  presents  one  simple  and  plausible  neural  cir¬ 
cuitry  for  divisive  normalization.  This  circuit  is  based 
on  Heeger's  model  of  gain  control  in  simple  cells,  where 
the  inhibitory  (possibly  of  shunting  type)  feedback  con¬ 
nections  perform  the  pool  normalization  [10].  With  a 
certain  choice  of  nonlinearities,  this  model  has  a  steady 
state  solution  that  is  close  to  Eqn.  3.  The  normaliza¬ 
tion  is  "close  enough"  in  the  sense  that  the  denominator 
may  contain  a  constant  (related  to  the  strength  of  shunt¬ 
ing  inhibition)  or  the  nonlinearity  may  not  exactly  be 
the  square  root  of  summed  squares  (see  Appendix  A). 

Another  crucial  operation  for  tuning  according  to 
Eqn.  2  is  the  scalar  product,  which  can  be  directly  ac¬ 
complished  by  the  synapses  (neglecting  dynamics).  In 
x-w,w  corresponds  to  a  vector  of  synaptic  weights,  and 
x  to  the  presynaptic  inputs,  as  shown  in  Fig.  lb. 

Combined  together,  the  circuits  in  Fig.  1  are  the  basic 
elements  for  a  network  that  can  compute  normalized 
scalar  product,  which  in  turn  would  produce  tuning  be¬ 
havior  in  a  general  multidimensional  input  space. 


4  Comparison  between  the  Gaussian 
function  and  the  normalized  scalar 
product 

In  this  section,  two  different  representations  of  tuning 
are  compared.  One  is  the  Gaussian  function,  based  on 
Euclidean  distance  measure,  and  the  other  is  based  on 
normalized  scalar  product  (NSP).  They  are  related  to 
each  other  by  Eqn.  2,  and  we  show  that  both  forms 
of  tuning  are  qualitatively  equivalent  and  can  be  made 
quantitatively  close. 

Mathematically,  the  Gaussian  tuning  function  and 
the  normalized  scalar  product  with  a  sigmoid  nonlin¬ 
earity  are  represented  as 


Rg  auss 

Rnsp 


0\x—w\2 /2a2 


1 


■  g  |£|+C  O 


(4) 

(5) 


The  sigmoid  is  a  commonly-used  transfer  function  for 
modeling  the  relationship  between  the  presynaptic  and 
postsynaptic  activations  or  membrane  depolarizations 
in  neurons.  It  sharpens  the  tuning  behavior  created  by 
normalized  scalar  product  and  allows  a  better  approx¬ 
imation  of  the  Gaussian  function,  as  the  parameters  a 
and  6  are  adjusted.^ 

'In  our  simulation,  a  nonlinear  fitting  routine  (nlinfitin 
Matlab)  was  used  to  find  the  best  a  and  f3  with  fixed  c  =  0.1 
in  Rnsp  for  a  given  w  and  a. 
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Figure  1:  Simple  neural  circuits  for:  (a)  Divisive  normalization,  y  =  x\ /(c  +  y  J2j  xj )  based  on  [10].  This  circuit 
is  just  one  possibility.  Divisive  normalization  may  be  computed  alternatively  by  feedforward  (instead  of  feedback) 
inhibition  at  the  dendrites  as  in  [18].  (b)  Scalar  product,  y  =  JA  xjw:i  =  x  ■  w.  (c)  Normalized  scalar  product.  This 
circuit  can  produce  Gaussian-like  tuning. 


For  Roauss /  w  specifies  the  center  of  the  Gaussian 
function  in  a  multidimensional  space.  The  Gaussian 
width  a  determines  the  sharpness  or  sensitivity  of  tun¬ 
ing  (er  need  not  be  the  same  along  different  dimensions). 
For  RnsP'W  specifies  the  direction  of  the  feature  vector 
along  which  the  response  is  maximal,  and  the  param¬ 
eters  a  and  (3  determine  the  sharpness  of  the  tuning. 
In  both  cases,  the  response  is  maximal  if  the  input  x  is 
matched  to  the  target  w. 

Fig.  2  shows  a  few  direct  comparisons  between 
Rcauss  and  Rnsp ■  Although  not  identical,  Rnsp  and 
Rcauss  exhibit  comparable  tuning  behaviors.  Because 
of  the  normalization,  the  dimensionality  of  Rnsp  is 
one  less  than  that  of  Rcauss ■  With  the  same  number 
of  afferents  n,  the  Gaussian  tuning  function  may  be 
centered  at  any  points  in  lZn,  whereas  the  normalized 
scalar  product  is  tuned  to  the  direction  of  the  vector  in 
Sn  or  7 Z"-1.  An  obvious  way  of  avoiding  such  limi¬ 
tation  is  to  assume  a  constant  dummy  input  and  to  in¬ 
crease  the  dimensionality  of  the  input  vector,  which  was 
the  approach  taken  here  as  in  [14].  Then,  the  normal¬ 
ized  scalar  product  may  be  tuned  to  any  arbitrary  vec¬ 
tor  w,  just  like  the  Gaussian  function  (see  Appendix  B 
for  more  discussions  on  this  issue). 

5  Discussion 

In  this  paper,  we  described  how  the  normalized  scalar 
product  can  account  for  the  tuning  of  neural  responses. 
We  also  sketched  a  plausible  neural  circuit. 

The  normalization  for  tuning  provides  some  new  in¬ 
sights  and  predictions.  For  example,  along  the  ventral 
pathway  of  primate  visual  cortex,  the  receptive  field 
size  on  average  increases,  and  neurons  show  tuning  to 
increasingly  complex  features  [11].  In  order  to  build  a 


larger  receptive  field  and  to  increase  feature  complex¬ 
ity,  the  neurons  may  be  pooling  from  many  afferents 
covering  different  parts  of  receptive  fields.  The  affer¬ 
ent  cells  within  the  pool  would  interact  via  normal¬ 
ization  operation,  whose  interaction  may  appear  as  a 
center-surround  effect  as  observed  in  VI  [5].  If  indeed 
a  general  mechanism  for  tuning,  normalization  would 
be  present  in  other  cortical  areas,  where  similar  center- 
surround  or  interference  effects  may  be  observable. 

The  effects  of  normalization  may  also  appear  when¬ 
ever  the  response  of  one  afferent  in  the  normalization 
pool  is  modulated  (for  example,  an  attentional  mecha¬ 
nism  through  a  feedback  connection).  Change  in  one 
neuron's  response  may  affect  not  only  the  output  of  the 
network,  but  the  response  of  other  afferent  neurons  in 
the  normalization  pool. 

We  also  note  that  this  scheme  for  cortical  tuning  has 
implications  for  learning  and  memory,  which  would  be 
accomplished  by  adjusting  the  synaptic  weights  accord¬ 
ing  to  the  activation  patterns  of  the  afferent  cells. 

Interestingly,  similar  neural  circuits  may  be  involved 
in  increasing  the  invariance  properties  of  neurons.  It 
has  been  observed  that  IT  neurons  show  certain  degree 
of  translation  and  scale  invariance  [2,  13],  and  so  do 
the  V4  neurons  [15].  One  way  of  producing  invariance 
is  the  maximum  operation,  whose  approximate  imple¬ 
mentation  may  involve  a  form  of  pool  normalization 
[26].  A  computational  model  [19,  20]  has  shown  that 
Gaussian-like  tuning  and  maximum  operations  were 
sufficient  to  capture  object  recognition  processing  in  vi¬ 
sual  cortex.  We  claim  here  that  similar  inhibitory  neural 
circuits  with  different  nonlinearities  (see  Appendix  C) 
may  accomplish  both  operations. 

In  the  past,  various  neural  micro-circuits  have  been 
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(a)  Comparison  in  1 -dimension 

y  =  e-(x-1  )2/20-22  y  =  e-<x-0-5)2/20-22 


(b)  Comparison  in  2-dimension 

RGauss  RNSP 


V 


0  0.5  1  0  0.5  1 


(c)  Comparison  in  higher  dimensions  ( d  =  5, 10,  20) 


w.  =  1  w.  =  random 


Figure  2:  Comparison  of  Raauss  and  Rnsp  in  several  dimensions.  Note  that  in  all  cases,  the  Gaussian  tuning 
function  can  be  approximated  by  the  normalized  scalar  product  followed  by  a  sigmoid  nonlinearity  The  parameters 
a  and  /3  in  the  sigmoid  are  found  with  nonlinear  fitting,  while  c  was  fixed  at  0.1.  As  pointed  out  in  Appendix  B, 
a  dummy  input  was  introduced  to  obtain  tuning  to  an  arbitrary  w  (i.e.,  Rnsp  is  in  <S"  +  1 ).  (a)  Comparison  in  1- 
dimension:  Raauss  (black)  with  a  =  0.2  and  Rnsp  (red)  are  shown  for  w  =  1  (left)  and  w  =  0.5  (right),  (b)  Similar 
comparisons  in  2-dimension:  w  =  (1, 1)  (top)  and  w  =  (0.4,  0.6)  (bottom),  (c)  Comparisons  in  higher  dimensions. 
Since  the  visualization  of  the  entire  function  is  difficult  for  high  dimensions,  1000  random  points  are  sampled  from 
the  space.  The  same  nonlinear  fitting  routine  was  used  to  find  the  parameters  in  Rnsp ■  The  width  of  Gaussian  is 
scaled  according  to  a  =  0.2 \fd,  where  d  is  the  dimensionality. 


proposed  to  implement  a  normalization  operation.  The 
motivation  was  to  account  for  gain  control.  We  make 
here  the  new  proposal  that  another  role  for  normalizing 
local  circuits  in  brain  is  to  provide  the  key  step  for  mul¬ 
tidimensional,  Gaussian-like  tuning.  In  fact  this  may 
be  the  main  reason  for  the  widespread  presence  of  gain 
control  circuits  in  cortex  where  tuning  to  optimal  stim¬ 
uli  is  a  common  property. 
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Appendix  A:  Normalization  circuit 

The  model  in  Fig.  1  is  based  on  [10],  and  the  steady-state  responses  of  the  neurons  (neglecting  dynamics)  are  deter¬ 
mined  by  the  following: 


°  -  ^ 
Xi '  r 


(6) 

(7) 


The  inhibitory  signal  G  depends  on  the  pooled  responses.  The  particular  choice  of  nonlinearity  (square  root  of 
summed  squares)  yields  a  mathematically  convenient  form  of  normalization.  Other  choices  can  produce  tuned 
responses,  although  they  are  not  as  easy  to  track  analytically.  The  response  Ri  is  proportional  to  the  input  Xi,  subject 
to  an  inhibitory  signal  operating  multiplicatively.  Such  multiplicative  interaction  may  arise  from  the  inhibition  of 
shunting  type,  as  noted  in  [24].  The  rectification  operation  for  ensuring  positive  neural  response  is  denoted  by  [  ]+. 
With  a  little  algebra, 

Ri  = - ^=,  (8) 

which  is  the  same  as  Eqn.  3,  except  for  the  positive  constant  c,  related  to  the  strength  of  inhibition.  Because  of  c,  the 
above  equation  is  not  the  true  normalization  in  a  mathematically  rigorous  sense,  but  as  shown  in  Appendix  B,  this 
approximate  normalization  is  enough  to  create  Gaussian-like  tuning. 

Finally, 

y  =  '^2iWiRi-’  (9) 

i 

resulting  in  normalized  scalar  product,  capable  of  producing  tuning  behavior. 
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Appendix  B:  Optimal  templates  in  normalized  scalar  product 

Since  the  scalar  product  x  ■  w  measures  the  cosine  of  the  angle  between  two  vectors,  the  maximum  occurs  when 
those  two  vectors  are  parallel.  Because  it  is  also  proportional  to  the  length  of  the  vector,  a  simple  scalar  product  is 
not  as  flexible  as  Gaussian  function  which  can  have  an  arbitrary  center.  We  may  assume  that  both  vectors  x  and 
w  are  normalized  (x  by  pool  normalization  and  w  by  Oja's  rule  [6],  for  example),  so  that  only  the  direction  within 
the  input  space  is  relevant.  However,  a  more  flexible,  simple  workaround  is  to  assume  a  constant  dummy  input, 
which  introduces  an  extra  dimension  and  allows  tuning  for  any  w  [14].  This  constant  may  be  the  resting  activity  of 
a  neuron. 

Using  the  result  of  derivation  from  the  previous  section  and  assuming  such  dummy  unit  (indexed  with  d  in  uj,j  and 
Xd),  the  response  of  the  normalizing  neural  circuit  is  given  by 


n 

y  Wj  xj + wd  xd 


n 

c+\ 

yx2j+xd 

\ 

1=1 

(10) 


which  can  be  viewed  as  a  normalized  scalar  product  in  (n  +  l)-dimension.  Then,  using  elementary  calculus,  it  is 
easy  to  verify  that  by  choosing  Wd  and  x,i,  the  maximum  response  occurs  when  x  -  w,  for  arbitrary  wt . 

Let's  take  the  partial  derivative: 


Settin§  iu  =  0' 


n 


dy 

Wi 

y  Wj  Xj + wd  xd 

1=1 

\  2  Xi 

dxi 

C+\ 

n 

y  + xd 

yx2j+x2d\  . 

n 

y x 2j + xd 

\ 

1=1 

v  \ 

U  )  \ 

1=1 

0  =  Wi 


hi 

yxj+x<i)  \ 

V  \ 

1=1  j  \ 

j=i 


X  X) 


Wj  Xj  +  Wd  Xd 


Setting  Xi  =  vjt ,  Vi  and  simplifying  the  expression. 


wd 


+  1  +  Xd- 


(11) 


(12) 


(13) 


As  long  as  the  above  condition  is  met,  any  arbitrary  w  can  serve  as  an  optimal  template,  and  since  «>,/  and  Xd  can  be 
freely  chosen,  it  is  easily  satisfied.  In  particular,  set  Xd  =  1  and  uhi  =  cy^E^i  w j  +  1  +  1/  as  done  in  the  simulations 
for  Fig.  2. 


Appendix  C:  Maximum-like  operation 


With  slightly  different  nonlinearities  in  normalization,  similar  gain  control  circuit  could  be  used  to  perform 
maximum-like  operation  on  the  inputs  to  a  neuron  [26].  Consider  the  following  divisive  normalization  (compare 
with  Eqn.  8): 


<7+1 


Ri  = 


■  +  E 


(14) 


For  sufficiently  high  q, 


Ri  = 


Xi,  if 

0,  if 


—  %max 


Xi  ^  X 


max 


(15) 


With  y  =  Ei  WiRi  where  uj,  -  1,  the  final  output  is  the  maximum  of  the  inputs. 


8 


